Plan: Merge-queue E2E (Kind + CloudNativePG + CoderTemplates)

Context / Why

We want an end-to-end test that can run in GitHub’s merge queue (merge_group) to ensure:

1. coder-k8s (controller) can reconcile a real CoderControlPlane backed by PostgreSQL.
2. The in-cluster aggregated API server is available.
3. We can use kubectl to create and retrieve CoderTemplate resources (served by the aggregated API server and backed by the Coder API).

This replaces the current Kind “smoke” test that only verifies a CoderControlPlane CR can be created.

Evidence (repo facts)

• Existing CI + current Kind smoke job: .github/workflows/ci.yaml (job e2e-kind).
• Current in-cluster e2e Deployment runs controller-only: config/e2e/deployment.yaml.
• Aggregated API server can be enabled in Kind using APIService.insecureSkipTLSVerify: true:
  • deploy/apiserver-service.yaml
  • deploy/apiserver-apiservice.yaml
• CloudNativePG example manifests already exist and are suitable to reuse:
  • examples/cloudnativepg/00-namespace.yaml
  • examples/cloudnativepg/cnpg-cluster.yaml
  • examples/cloudnativepg/codercontrolplane.yaml
• Aggregated API server selects a CoderControlPlane in the request namespace and reads operator URL/token from status+Secret (Explore report: “CoderTemplate Aggregated API Usage Guide”).
• CoderTemplate create path can upload spec.files as a zip, create a template version, then create the template (no synchronous wait in codersdk): internal/aggregated/storage/template.go.

Implementation details

1) Enable aggregated API server in the e2e Deployment

File: config/e2e/deployment.yaml

Change e2e deployment from controller-only to --app=all and expose the HTTPS port used by the APIService.

Proposed diff shape:

containers:
  - name: manager
    image: ghcr.io/coder/coder-k8s:e2e
    args: ["--app=all"] # was --app=controller
    ports:
      - containerPort: 8081
        name: health
      - containerPort: 6443
        name: https

Rationale: the deploy/apiserver-service.yaml targets port 6443 on pods labeled app: coder-k8s.

2) Add an E2E CoderTemplate manifest

New file: config/e2e/codertemplate.yaml

Create a minimal template in the same namespace as the E2E CoderControlPlane (the existing CNPG example uses namespace: coder).

apiVersion: aggregation.coder.com/v1alpha1
kind: CoderTemplate
metadata:
  name: default.e2e-template
  namespace: coder
spec:
  organization: default
  displayName: "E2E Template"
  description: "Created by coder-k8s merge-queue e2e"
  files:
    main.tf: |
      terraform {
        required_version = ">= 1.0"
      }
      resource "null_resource" "example" {}

Notes:

• metadata.name must be <org>.<template-name> and spec.organization must match the <org> prefix (enforced server-side).
• Keeping the template minimal avoids turning this into a workspace/provisioner E2E.

3) Replace the CI Kind smoke job with full CNPG + templates E2E

File: .github/workflows/ci.yaml

3.1 Keep the job identity stable

Keep the job id e2e-kind (and ideally keep the name: similar) so that existing branch protection / required checks don’t unexpectedly change.

3.2 Runtime control: run full E2E in merge queue only

Full CNPG + Coder bring-up will be materially slower than the current smoke test.

Recommended approach:

• Keep the e2e-kind job running on PRs (so the check exists), but make it a fast no-op.
• Run the full E2E only when github.event_name == 'merge_group'.

This satisfies “only run it in the merge queue” while still allowing the check to be configured as required.

Implementation pattern:

  e2e-kind:
    name: E2E (Kind + CNPG + Templates)
    ...
    steps:
      - name: Skip full E2E outside merge queue
        if: github.event_name != 'merge_group'
        run: |
          echo "Skipping full E2E on ${GITHUB_EVENT_NAME}; it runs in merge queue (merge_group)."

      - name: Checkout
        if: github.event_name == 'merge_group'
        uses: actions/checkout@...

      # all heavy steps also gated on merge_group

If you do want some signal on PRs, a compromise is to keep the old smoke steps under pull_request and the full E2E under merge_group.

3.3 Full merge-queue E2E steps (merge_group only)

Replace the existing “apply sample CR and verify it exists” with:

1. Create Kind cluster (existing)
2. Build + load ghcr.io/coder/coder-k8s:e2e image (existing)
3. Apply base manifests (existing):
   • kubectl apply -f config/e2e/namespace.yaml
   • kubectl apply -f config/crd/bases/
   • kubectl apply -f config/rbac/
4. Enable aggregated API (new):
   • kubectl apply -f deploy/apiserver-service.yaml
   • kubectl apply -f deploy/apiserver-apiservice.yaml
5. Deploy coder-k8s (updated config/e2e/deployment.yaml)
6. Wait for operator:
   • kubectl wait --for=condition=Available deploy/coder-k8s -n coder-system --timeout=120s
   • kubectl wait --for=condition=Available apiservice/v1alpha1.aggregation.coder.com --timeout=180s
7. Install CloudNativePG operator (pinned version) + wait:
   • kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.25/releases/cnpg-1.25.0.yaml
   • kubectl wait --for=condition=Available deploy/cnpg-controller-manager -n cnpg-system --timeout=180s
8. Provision Postgres + CoderControlPlane using existing repo examples:
   • kubectl apply -f examples/cloudnativepg/
   • kubectl -n coder wait --for=condition=Ready cluster/coder-db --timeout=10m
9. Wait for Coder to come up and for operator access bootstrap:
   • kubectl -n coder rollout status deployment/coder --timeout=10m
   • kubectl -n coder wait --for=jsonpath='{.status.phase}'=Ready codercontrolplane/coder --timeout=10m
   • kubectl -n coder wait --for=jsonpath='{.status.operatorAccessReady}'=true codercontrolplane/coder --timeout=10m
   • Verify the token Secret exists:

     TOKEN_SECRET=$(kubectl -n coder get codercontrolplane coder -o jsonpath='{.status.operatorTokenSecretRef.name}')
     test -n "$TOKEN_SECRET"
     kubectl -n coder get secret "$TOKEN_SECRET"

10. Create + retrieve a CoderTemplate via kubectl (new):
    • kubectl apply -f config/e2e/codertemplate.yaml
    • kubectl -n coder get codertemplates
    • kubectl -n coder get codertemplate default.e2e-template -o yaml

3.4 Add failure diagnostics (high value for flake triage)

Add if: failure() steps to dump state:

• kubectl get pods -A
• kubectl describe apiservice v1alpha1.aggregation.coder.com
• kubectl -n coder-system logs deploy/coder-k8s --tail=200
• kubectl -n coder describe codercontrolplane coder
• kubectl -n coder logs deploy/coder --tail=200 (best-effort)
• kubectl -n cnpg-system logs deploy/cnpg-controller-manager --tail=200

4) Make it required for merge queue completion

This is a repo settings change (not a code change): mark the check run name (e.g., E2E (Kind + CNPG + Templates)) as a required status check on the protected branch that uses merge queue.

Keep the job id/name stable to avoid having to reconfigure required checks.

Validation plan (when implementing)

• Manually run the new job steps locally once (optional) using hack/kind-dev.sh + kubectl apply commands.
• In CI, confirm on a test PR:
  • PR event: e2e-kind completes quickly (skipped heavy steps) and is green.
  • Merge queue (merge_group): full E2E runs, creates CoderControlPlane + CoderTemplate, and succeeds.